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Reactive Reasoning with the Event Calculus 

Alexander Artikis^ and Marek Sergot^ and Georgios Paliouras^ 


Abstract. Systems for symbolic event recognition accept as input a 
stream of time-stamped events from sensors and other computational 
devices, and seek to identify high-level composite events, collections 
of events that satisfy some pattern. RTEC is an Event Calculus dialect 
with novel implementation and ‘windowing’ techniques that allow 
for efficient event recognition, scalable to large data streams. RTEC 
can deal with applications where event data arrive with a (variable) 
delay from, and are revised by, the underlying sources. RTEC can up¬ 
date already recognised events and recognise new events when data 
arrive with a delay or following data revision. Our evaluation shows 
that RTEC can support real-time event recognition and is capable of 
meeting the performance requirements identified in a recent survey 
of event processing use cases. 

1 Introduction 

Systems for symbolic event recognition (‘event pattern matching’) 
accept as input a stream of time-stamped simple, derived events 
(SDE)s. A SDE (‘low-level event’) is the result of applying a com¬ 
putational derivation process to some other event, such as an event 
coming from a sensor [21]. Using SDEs as input, event recognition 
systems identify composite events (CE)s of interest—collections of 
events that satisfy some pattern. The ‘definition’ of a CE (‘high-level 
event’) imposes temporal and, possibly, atemporal constraints on its 
subevents, i.e. SDEs or other CEs. Consider e.g. the recognition of 
attacks on computer network nodes given the TCP/IP messages. 

Numerous recognition systems have been proposed in the litera¬ 
ture [10]. Recognition systems with a logic-based representation of 
CE definitions, in particular, have recently been attracting attention 
[4]. They exhibit a formal, declarative semantics, in contrast to other 
types of recognition system that usually rely on an informal and/or 
procedural semantics. However, non-logic-based CE recognition sys¬ 
tems have proven to be, overall, more efficient than logic-based ones. 
To address this issue, we present an efficient dialect of the Event Cal¬ 
culus [18], called ‘Event Calculus for Run-Time reasoning’ (RTEC). 
The Event Calculus is a logic programming formalism for repre¬ 
senting and reasoning about events and their effects. RTEC includes 
novel implementation techniques for efficient CE recognition, scal¬ 
able to large SDE and CE volumes. A set of interval manipulation 
constructs simplify CE definitions and improve reasoning efficiency. 
A simple indexing mechanism makes RTEC robust to SDEs that are 
irrelevant to the CEs we want to recognise and so RTEC can operate 
without SDE filtering modules. Finally, a ‘windowing’ mechanism 
supports real-time CE recognition. One main motivation for RTEC 
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is that it should remain efficient and scalable in applications where 
SDEs arrive with a (variable) delay from, or are revised by, the un¬ 
derlying SDE detection system: RTEC can update the already recog¬ 
nised CEs, and recognise new CEs, when SDEs arrive with a delay 
or following revision. The code of RTEC is available at <http: 
//users.lit.demokritos.gr/~a.artikis/EC.html>. 

We evaluate RTEC on public space surveillance from video con¬ 
tent. In this application, the SDEs are the ‘short-term activities’ de¬ 
tected on video frames—e.g. a person walking, running or being in¬ 
active. The aim then is to recognise ‘long-term activities’, i.e. short¬ 
term activity combinations, such as when a person leaves an object 
unattended, when two people are moving together, when they are 
having a meeting or fighting. The CE definitions are quite complex, 
allowing for a realistic evaluation of the efficiency of RTEC. This is 
in contrast to the majority of related approaches where rather simple 
CE definitions are used for empirical analysis. Our evaluation shows 
that RTEC supports real-time CE recognition and is capable of meet¬ 
ing the performance requirements of most of today’s applications as 
estimated by a recent survey of event processing use cases [5]. 

The remainder of the paper is structured as follows. Sections 2 
and 3 present the expressivity of RTEC and the way it performs rea¬ 
soning. The experimental evaluation is given in Section 4. Section 5 
summarises the presented work, puts the work in context, and out¬ 
lines directions for further research. 

2 Event Calculus 

Our system for CE recognition is based on an Event Calculus dialect. 
The Event Calculus [18] is a logic programming formalism for rep¬ 
resenting and reasoning about events and their effects. For the dialect 
introduced here, called RTEC, the time model is linear and includes 
integer time-points. Variables start with an upper-case letter, while 
predicates and constants start with a lower-case letter. Where f is a 
fluent —a property that is allowed to have different values at different 
points in time—the term F = V denotes that fluent F has value V. 
Boolean fluents are a special case in which the possible values are 
true and false. holdsAt(F = V, T) represents that fluent F has value 
U at a particular time-point T. holdsFor(T’ = V, I) represents that I 
is the list of the maximal intervals for which F — V holds continu¬ 
ously. holdsAt and holdsFor are defined in such a way that, for any 
fluent F, holdsAt)!^ = V, T) if and only if T belongs to one of the 
maximal intervals of I for which holdsFor)!'’ — V,I). 

An event description in RTEC includes rules that define the event 
instances with the use of the happensAt predicate, the effects of 
events with the use of the InitiatedAt and terminatedAt predicates, and 
the values of the fluents with the use of the holdsAt and holdsFor 
predicates, as well as other, possibly atemporal, constraints. Table 1 
summarises the RTEC predicates available to the event description 
developer. The last three items in the table are interval manipulation 



predicates specific to RTEC. 

Table 1: Main predicates of RTEC. 


Predicate 

Meaning 

happensAt(P, T) 

Event E occurs at time T 

holdsAt(P = U,T) 

The value of fluent FisVat time T 

holdsFor(P = U,/) 

I is the list of the maximal intervals 
for which F = V holds continuously 

initiatedAt(P = U, T) 

At time T a period of time for which 

F = V is initiated 

terminatedAt(P = V, T) 

At time T a period of time for which 

F = V is terminated 

relative- 

complement 

a\\.(I',L,I) 

I is the list of maximal intervals produced 
by the relative complement of the list 
of maximal intervals I' with respect to 
every list of maximal intervals of list L 

union_all(L, I) 

I is the list of maximal intervals 
produced by the union of the lists of 
maximal intervals of list L 

intersect_all(L, I) 

I is the list of maximal intervals 
produced by the intersection of 
the lists of maximal intervals of list L 


We represent instantaneous SDEs and CEs by means of happen- 
sAt, while durative SDEs and CEs are represented as fluents. The 
majority of CEs are durative and, therefore, in CE recognition the 
task generally is to compute the maximal intervals for which a fluent 
representing a CE has a particular value continuously. 

2,1 Simple Fluents 

Fluents in RTEC are simple or statically determined. We assume, 
without loss of generality, that these types are disjoint. For a sim¬ 
ple fluent F, F — V holds at a particular time-point T if F — V has 
been initiated by an event that has occurred at some time-point ear¬ 
lier than T, and has not been terminated at some other time-point 
in the meantime. This is an implementation of the law of inertia. To 
compute the intervals I for which F = V, i.e. holdsFor(T' = V, I), 
we find all time-points Ta at which F = V is initiated, and then, for 
each Ts, we compute the first time-point Tf after Ts at which F = V 
is ‘broken’. The time-points at which F = V is initiated are com¬ 
puted by means of domain-specific initiatedAt rules. The time-points 
at which F = V is ‘broken’ are computed as follows: 

broken(F = V, Ts, T) ^ 

terminatedAt(T’ = F, Tf), Ts < Tf < T ^ ^ 

broken(F = 'Ei, Ts, T) ^ 

initiatedAt(F = y2, Tf), Ts <Tf < T, Fi ^ Fa ^ ’ 

broken(T' = V, Ts,T) represents that a maximal interval starting at 
Ts for which F = V holds continuously is terminated at some time 
Tf such that Ts<Tf<T. Similar to initiatedAt, terminatedAt rules are 
domain-specific (examples are presented below). According to rule 
(2), if F = Va is initiated at T/ then effectively F = Vi is terminated 
at time Tf, for all other possible values Vj of F. Rule (2) ensures 
therefore that a fluent cannot have more than one value at any time. 
We do not insist that a fluent must have a value at every time-point. 
There is a difference between initiating a Boolean fluent F = false 
and terminating F — true: the former implies, but is not implied by, 
the latter. 


RTEC stores and indexes holdsFor intervals as they are computed 
for any given fluent-value F = V: thereafter intervals for F = V 
are retrieved from the computer memory without the need for re¬ 
computation. Similarly, a holdsAt query for F = V looks up F’s 
value in the holdsFor cache. 

In public space surveillance, it is often required to detect when a 
person leaves an object unattended. Typically, an object carried by a 
person is not tracked by the computer vision algorithms—only the 
person that carries it is tracked. The object will be tracked, i.e. it will 
‘appear’, if and only if the person leaves it somewhere. Moreover, 
objects (as opposed to persons) can exhibit only inactive activity. Ac¬ 
cordingly, we define a durative ‘leaving an object’ CE as follows: 

\n\l\aie6At(leaving_object(P, Obj) =[rue, T) ■(— 
happensAi)appear(Obj), T), 
holdsAt(mocfwe(06)) =true, T), (3) 

holdsAt(dose(P, Obj)=true, T), 
Uo\6sA\.{person{P) — true, T) 

\n\t\aie6At(leaving_object(P, Obj) =ia\se, T) ■(— 
happensAt{disappear{Obj), T) 

In rule (3) leaving-object{P, Obj) =true is initiated at time T if 
Obj ‘appears’ at T, it is inactive at T, and there is a person P ‘close’ 
to Obj at T. appear and inactive are instantaneous SDE and du¬ 
rative SDE respectively. SDE are detected on video frames in this 
application. close{A, B) is true when the distance between A and B 
does not exceed some threshold of pixel positions. 

There is no explicit information about whether a tracked entity 
is a person or an inanimate object. We define the simple fluent 
person{P) to have value true if P has been active, walking, running 
or moving abruptly since P first ‘appeared’. The value of person(P) 
has to be time-dependent because the identifier P of a tracked entity 
that ‘disappears’ (is no longer tracked) at some point may be used 
later to refer to another entity that ‘appears’ (becomes tracked), and 
that other entity may not necessarily be a person. This is a feature of 
the application and not something that is imposed by RTEC. 

Unlike the specification of person, it is not clear from the data 
whether a tracked entity is an object. person{P) = false does not 
necessarily mean that P is an object; it may be that P is not tracked, 
or that P is a person that has never walked, run, been active or moved 
abruptly. Note finally that rule (3) incorporates a reasonable simpli¬ 
fying assumption, that a person entity will never exhibit ‘inactive’ 
activity at the moment it first ‘appears’ (is tracked). If an entity is 
‘inactive’ at the moment it ‘appears’ it can be assumed to be an ob¬ 
ject, as in the first two conditions of rule (3). 

Rule (4) expresses the conditions in which leaving_object ceases 
to be recognised, leaving .object [P, Obj) becomes false when 
the object in question is picked up. An object that is picked 
up by someone is no longer tracked—it ‘disappears’—terminating 
leaving .object, {disappear is an instantaneous SDE.) The maximal 
intervals during which leaving.object{P, Obj) =true holds contin¬ 
uously are computed using the built-in RTEC predicate holdsFor 
from rules (3) and (4). 

Consider another example from public space surveillance: 

initiatedAt(moOTn(;(Pj, Pg) =true, T) ■(— 
happensAt(start(w)a/A:m 3 (Pj) =true), T), 
\r\o\dsAi{walking{Ps) =irue, T), 
holdsAt(dose(Pj, Pg) =true, T) 



list L, as, e.g.: 


initiatedAt(moOTn 5 (Pj, Pg) =true, T) t— 

happensAt(start(t«a/fcing(Pg) =true), T), 
ho\dsA\{walking{Pi ) = true, T), 
holdsAt(c/ose(Pj, Pg) =true, T) 

initiatedAt(mown 5 (Pj, Pg) =true, T) t— 

happensAt(start( close (Pi, P 2 )= true), T), 
holdsAt(t«oZfcmg(Pj) = true, T), 
holdsAt(t«aZfcmsi(Pg) =true, T) 

terminatedAt(moOTn 5 (Pi, Pg) = true, T) t— 
happensAt(end(u)oZfcm(;(Pj) =true), T) 

terminatedAt(moOTn 5 (Pi, Pg) = true, T) t— 
happensAt(end(woZfcm(;(Pg) =true), T) 

terminatedAt(moOTn 5 (Pi, Pg) = true, P) t— 
happensAt(end(cZose(Pi, Pg) = true), T) 


( 6 ) 


(7) 

( 8 ) 
(9) 

( 10 ) 


walking is a durative SDE detected on video frames. start(P = V) 
(resp. end(P = V)) is a built-in RTEC event taking place at each 
starting (ending) point of each maximal interval for which F = V 
holds continuously. The above formalisation states that Pi is moving 
with Pg when they are walking close to each other. 

One of the main attractions of RTEC is that it makes available 
the power of logic programming to express complex temporal and 
atemporal constraints, as conditions in initiatedAt and terminatedAt 
rules for durative CEs, and happensAt rules for instantaneous CEs. 
E.g. standard event algebra operators, such as sequence, disjunction, 
parallelism, etc, may be expressed in a RTEC event description. 


2.2 Statically Determined Fluents 

In addition to the domain-independent dehnition of holdsFor, an 
event description may include domain-specific holdsFor rules, used 
to define the values of a fluent F in terms of the values of other flu¬ 
ents. We call such a fluent F statically determined. holdsFor rules of 
this kind make use of interval manipulation constructs—see the last 
three items of Table 1. Consider, e.g. moving as in rules (5)-(10) but 
defined instead as a statically determined fluent: 

holdsFor(moOTn 5 i(Pj, Pg) =true, /) t— 
holdsFor(M)aZA:in 5 (Pj) =true, P), 
holdsFor(M)aZA:m 5 (Pg) =true, /g), (11) 

holdsFor(cZose(Pj, Pg) =true, I 3 ), 
lntersect_all([P, Jg,/s], I) 

The list I of maximal intervals during which Pi is moving with Pg 
is computed by determining the list li of maximal intervals during 
which Pi is walking, the list /g of maximal intervals during which Pg 
is walking, the list p of maximal intervals during which Pi is close 
to Pg, and then calculating the list I representing the intersections of 
the maximal intervals in li, Jg and Jg. 

RTEC provides three interval manipulation constructs: union.all, 
lntersect_all and relative_complement_all. union_all(L, /) computes 
the list I of maximal intervals representing the union of maximal 
intervals of the lists of list P. Eor instance: 

union_all([[(5, 20), (26,30)], [(28, 35)]], [(5, 20), (26,35)]) 

A term of the form (Ts, Te) in RTEC represents the closed-open 
interval [Ts, Pe). I in union.all(L, 7) is a list of maximal intervals 
that includes each time-point that is part of at least one list of L. 

intersect_all(L, 7) computes the list 7 of maximal intervals such 
that 7 represents the intersection of maximal intervals of the lists of 


intersect_all([[(26, 31)], [(21, 26), (30, 40)]], [(30, 31)]) 

7 in intersect_all(P, 7) is a list of maximal intervals that includes each 
time-point that is part of all lists of L. 

relative_complement_all(7', P, 7) computes the list 7 of maximal 
intervals such that 7 represents the relative complements of the list 
of maximal intervals F with respect to the maximal intervals of the 
lists of list P. Below is an example of relative.complement.all: 

relative_complement_all([(5, 20), (26, 50)], 

[[(1,4), (18, 22)], [(28, 35)]], [(5,18), (26, 28), (35, 50)]) 

7 in relative_complement_all(7', P, 7) is a list of maximal intervals 
that includes each time-point of I' that is not part of any list of P. 

When defining a statically determined fluent F we will often want 
to say that, for all time-points T, F = V holds at T if and only if W 
holds at T where W is some Boolean combination of fluent-value 
pairs. RTEC provides optional shorthands for writing such defini¬ 
tions concisely. For example, the definition 


G = y iff 

(A = Vjor73 = I/g), 

(A = I//or B = 1/2'), ^ ' 

not G = Vs 

is expanded into the following holdsFor rule: 

holdsFor(G = V) 7) t— 

holdsFor(A = Ti, h), holdsFor(73 = Vg, p), 
union_all([7i,7g], h), 

holdsFor(A = I/i', h), holdsFor(B = Vg', h), 
union_all([74, 75 ], h), 
lntersect_all([73,76], I 7 ), 
holdsFor(G = V 3 , h), 
relative.complement.all ( 77 , [78], 7) 

The required transformation takes place automatically when event 
descriptions are loaded into RTEC. 

For a wide range of fluents, the use of interval manipulation 
constructs leads to a much more concise definition than the tradi¬ 
tional style of Event Calculus representation, i.e. identifying the var¬ 
ious conditions under which the fluent is initiated and terminated 
so that maximal intervals can then be computed using the domain- 
independent holdsFor. Compare, e.g. the statically determined and 
simple fluent representations of moving in rules (11) and (5)-(10) 
respectively. 

The interval manipulation constructs of RTEC can also lead to 
much more efficient computation. The complexity analysis may be 
found in [3]. 

2.3 Semantics 

CE definitions are (locally) stratified logic programs [25]. We restrict 
attention to hierarchical definitions, those where it is possible to de¬ 
fine a function level that maps all fluent-values F = V and all events 
to the non-negative integers as follows. Events and statically deter¬ 
mined fluent-values 7^ = ’C of level 0 are those whose happensAt 
and holdsFor definitions do not depend on any other events or flu¬ 
ents. In CE recognition, they represent the input SDEs. There are 
no fluent-values 7^ = ’C of simple fluents F in level 0. Events and 
simple fluent-values of level n are defined in terms of at least one 



event or fluent-value of level n—1 and a possibly empty set of events 
and fluent-values from levels lower than n—1. Statically determined 
fluent-values of level n are defined in terms of at least one fluent- 
value of level n—1 and a possibly empty set of fluent-values from 
levels lower than n—1. Note that fluent-values F = Vi and F — Vj 
for Vi^Vj could be mapped to different levels. For simplicity how¬ 
ever, and without loss of generality, a fluent F itself is either simple 
or statically determined but not both. The CE definitions of public 
space surveillance, i.e. the holdsFor definitions of statically deter¬ 
mined fluents, initiatedAt and terminatedAt definitions of simple flu¬ 
ents and happensAt definitions of events, are available with the RTEC 
code. 

3 Run-Time Recognition 

CE recognition has to be efficient enough to support real-time 
decision-making, and scale to very large numbers of SDEs and CEs. 
SDEs may not necessarily arrive at the CE recognition system in a 
timely manner, i.e. there may be a (variable) delay between the time 
at which SDEs take place and the time at which they arrive at the CE 
recognition system. Moreover, SDEs may be revised, or even com¬ 
pletely discarded in the future, as in the case where the parameters of 
a SDE were originally computed erroneously and are subsequently 
revised, or in the case of retraction of a SDE that was reported by 
mistake, and the mistake was realised later [1]. Note that SDE re¬ 
vision is not performed by the CE recognition system, but by the 
underlying SDE detection system. 

RTEC performs CE recognition by computing and storing the 
maximal intervals of fluents and the time-points in which events oc¬ 
cur. CE recognition takes place at specified query times Qi,Q2, ■ ■ ■ ■ 
At each Qi the SDEs that fall within a specified interval—the ‘work¬ 
ing memory’ {WM) or ‘window’—are taken into consideration. All 
SDEs that took place before or at Qi — WM are discarded. This is to 
make the cost of CE recognition dependent only on the WM size and 
not on the complete SDE history. The WM size, and the temporal dis¬ 
tance between two consecutive query times — the ‘step’ (Qi—Qi-i) 
— are set by the user. 

At Qi, the maximal intervals computed by RTEC are those that can 
be derived from SDEs that occurred in the interval {Qi — WM, Qi], as 
recorded at time Qi. When WM is longer than the inter-query step, 
i.e., when Qi — WM<Qi-i<Qi, it is possible that an SDE occurs in 
the interval {Qi — WM, Qi-i] but arrives at RTEC only after Qi-i; 
its effects are taken into account at query time Qi. And similarly 
for SDEs that took place in {Qi — WM, Qi-i] and were subsequently 
revised after Qi-i. In the common case that SDEs arrive at RTEC 
with delays, or there is SDE revision, it is preferable therefore to 
make WM longer than the inter-query step. Note that information 
may still be lost. Any SDEs arriving or revised between Qi-\ and 
Qi are discarded at Qi if they took place before or at Qi — WM. To 
reduce the possibility of losing information, one may increase the 
WM size. Doing so, however, decreases recognition efficiency. 

Eigure 1 illustrates windowing in RTEC. In this example we have 
WM>Qi—Qi-i. To avoid clutter. Figure 1 shows streams of only 
five SDEs. These are displayed below WM, with dots for instanta¬ 
neous SDEs and lines for durative ones. Eor the sake of the example, 
we are interested in recognising just two CEs: 

• CEs, represented as a simple fluent (see Section 2.1). The starting 

and ending points, and the maximal intervals of CEs are displayed 

above WM in Figure 1. 

• CEstd, represented as a statically determined fluent (see Section 

2.2). For the example, the maximal intervals of CEstd are defined 
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Figure 1: Windowing in RTEC. 


to be the union of the maximal intervals of the two durative SDEs 
in Figure 1. The maximal intervals of CEstd are displayed above 
the CEs intervals. 

For simplicity, we assume that both CEs and CEstd are defined only 
in terms of SDE, i.e. they are not defined in terms of other CEs. 

Eigure 1 shows the steps that are followed in order to recognise 
CEs at an arbitrary query time, say Qias. Figure 1(a) shows the 
state of RTEC as computation begins at Qiss. All SDEs that took 
place before or at Q 137 — WM were retracted at Q 137 . The thick lines 
and dots represent the SDEs that arrived at RTEC between Q 137 and 
Qi3S', some of them took place before Q 137 . Figure 1(a) also shows 
the maximal intervals for the CE fluents CEs and CEstd that were 
computed and stored at < 5137 . 

The CE recognition process at Qiag considers the SDEs that took 
place in (Qisg —WM, Qiag], All SDEs that took place before or 
at Qi 3 g —WM are discarded, as shown in Eigure 1(b). For durative 
SDEs that started before Qigg —WM and ended after that time, RTEC 
retracts the sub-interval up to and including Q 133 — WM. Figure 1(b) 
shows the interval of a SDE that is partially retracted in this way. 

Now consider CE intervals. At Qi some of the maximal intervals 
computed at Qi-\ might have become invalid. This is because some 
SDEs occurring in {Qi — WM, Qi-i] might have arrived or been re¬ 
vised after Qi-i: their existence could not have been known at Qi_i. 
Determining which CE intervals should be (partly) retracted in these 
circumstances can be computationally very expensive. See Section 5 
for a discussion. We find it simpler, and more efficient, to discard all 
CE intervals in {Qi — WM, Qi] and compute all intervals from scratch 
in that period. CE intervals that have ended before or at Qi — WM are 
discarded. Depending on the user requirements, these intervals may 
be stored in a database for retrospective inspection of the activities 
of a system. 

In Figure 1(b), the earlier of the two maximal intervals computed 
for CEstd at Q137 is discarded at Qigg since its endpoint is before 





Qisg —WM. The later of the two intervals overlaps Qi^g — WM (an 
interval ‘overlaps’ a time-point t if the interval starts before or at 
t and ends after or at that time) and is partly retracted at Qiag. Its 
starting point could not have been affected by SDEs arriving between 
Q\3S, — WM and Qigg but its endpoint has to be recalculated. Accord¬ 
ingly, the sub-interval from Qigg — VTM is retracted at Qigg. 

In this example, the maximal intervals of CEm are determined 
by computing the union of the maximal intervals of the two dura- 
tive SDEs shown in Figure 1. At Qigg, only the SDE intervals in 
(Qiss —VFM, Qigg] are considered. In the example, there are two 
maximal intervals for CEsu in this period as can be seen in Fig¬ 
ure 1(c). The earlier of them has its startpoint at Qigg —VTM. Since 
that abuts the existing, partially retracted sub-interval for CEm 
whose endpoint is Qigg —VTM, those two intervals are amalgamated 
into one continuous maximal interval as shown in Figure 1(c). In this 
way, the endpoint of the CEatd interval that overlapped Qigg —VTM 
at Qi 37 is recomputed to take account of SDEs available at Qigg. (In 
this particular example, it happens that the endpoint of this interval 
is the same as that computed at Q 137 . That is merely a feature of this 
particular example. Had CEstd been defined e.g. as the intersection 
of the maximal intervals of the two durative SDE, then the intervals 
of CEstd would have changed in {Qi^g — WM, Qisr].) 

Eigure 1 also shows how the intervals of the simple fluent CEs 
are computed at Qisg. Arrows facing upwards (downwards) denote 
the starting (ending) points of CEs intervals. First, in analogy with 
the treatment of statically determined fluents, the earlier of the two 
CEs intervals in Figure 1(a), and its start and endpoints, are re¬ 
tracted. They occur before Qiag —VFM. The later of the two intervals 
overlaps Qiss — WM. The interval is retracted, and only its starting 
point is kept; its new endpoint, if any, will be recomputed at Qiag. 
See Figure 1(b). For simple fluents, it is simpler, and more efficient, 
to retract such intervals completely and reconstruct them later from 
their start and endpoints by means of the domain-independent holds- 
For rules, rather than keeping the sub-interval that takes place before 
Qiss — WM, and possibly amalgamating it later with another interval, 
as we do for statically determined fluents. 

The second step for CEs at Qi 3 g is to calculate its starting 
and ending points by evaluating the relevant initiatedAt and termi- 
natedAt rules. For this, we only consider SDEs that took place in 
{Qi3S — WM, Qi 3 g]. Eigure 1(c) shows the starting and ending points 
of CEs in {Qiss — WM, Qiag]. The last ending point of CEs that was 
computed at Q 137 was invalidated in the light of the new SDEs that 
became available at Q 138 (compare Eigures l(c)-(a)). Moreover, an¬ 
other ending point was computed at an earlier time. 

Einally, in order to recognise CEs at Q 138 we use the domain- 
independent holdsFor to calculate the maximal intervals of CEs 
given its starting and ending points. The later of the two CEs inter¬ 
vals computed at Q 137 became shorter when re-computed at Qisg. 
The second interval of CEs at Q 138 is open: given the SDEs avail¬ 
able at Qi 38 , we say that CEs holds since time t, where t is the last 
starting point of CEg . 

The discussion above showed that, when SDEs arrive with a vari¬ 
able delay, CE intervals computed at an earlier query time may be 
(partly) retracted at the current or a future query time. (And sim¬ 
ilarly if SDEs are revised.) Depending on the application require¬ 
ments, RTEC may be set to report: 

• CEs as soon as they are recognised, even if their intervals may be 

(partly) retracted in the future. 

• CEs whose intervals may be partly, but not completely, retracted 

in the future, i.e. CEs whose intervals overlap Qi+i — WM. 


• CEs whose intervals will not be even partly retracted in the future, 

i.e. CEs whose intervals end before or at Qt+i — WM. 

The example used for illustration shows how RTEC performs CE 
recognition. To support real-time reasoning, at each query time Qi all 
SDEs that took place before or at Qi — WM are discarded. To handle 
efficiently delayed SDEs and SDE revision, CE intervals within WM 
are computed from scratch. At Qt, the computed maximal CE inter¬ 
vals are those that can be derived from SDEs that occurred in the 
interval {Qi — WM, Qi], as recorded at time Qi. For completeness, 
RTEC amalgamates the computed intervals to any intervals ending at 
Qi — WM. More details about CE recognition in RTEC may be found 
at [3]. 

4 Experimental Results 

We present experimental results on the public space surveillance 
application. The experiments were performed on a computer with 
eight Intel i7 950@3.07GHz processors and 12GiB RAM, running 
Ubuntu Linux 12.04 and YAP Prolog 6.2.2. Each CE recognition 
time displayed in this section is the average of 30 runs. We use the 
CAVIAR benchmark dataset consisting of 28 surveillance videos of 
a public space <http: / /groups . inf . ed. ac. uk/vision/ 
CAVIAR/CAVIARDATA1>. The videos are staged—actors walk 
around, sit down, meet one another, leave objects behind, etc. Each 
video has been manually annotated by the CAVIAR team in order 
to provide the ground truth for ‘short-term activities’, i.e. activities 
taking place in a short period of time detected on individual video 
frames. (The frame rate in CAVIAR is 40 ms.) The short-term ac¬ 
tivities of CAVIAR concern an entity (person or object) entering or 
exiting the surveillance area, walking, running, moving abruptly, be¬ 
ing active or inactive. The CAVIAR team has also annotated the 28 
videos with ‘long-term activities’: a person leaving an object unat¬ 
tended, two people meeting, moving together and fighting. Short¬ 
term activities can be viewed as SDEs while long-term activities can 
be viewed as CEs. Consequently, the input to RTEC in this case study 
includes the set of annotated short-term activities, and the output is 
a set of recognised long-term activities. The CE definitions and the 
datasets on which the experiments were performed are available with 
the RTEC code. 

CE recognition for multiple pairs of entities. Figure 2(a) shows 
the results of experiments concerning all 45 pairs of the 10 entities 
tracked in the CAVIAR dataset. (In CAVIAR each CE concerns a pair 
of entities.) On average, 179 SDEs are detected per sec. We used a 
single processor for CE recognition concerning all 45 tracked pairs. 
That requires computing and storing the intervals of 645 CEs. We 
varied WM from 10 sec («2,000 SDEs) to 110 sec («19,000 SDEs). 
The inter-query step is set to 5 sec («1,000 SDEs). In all settings 
shown in Figure 2(a), RTEC performs real-time CE recognition. 

Larger datasets. We constructed a larger dataset by taking ten 
copies of the original CAVIAR dataset with new identifiers for the 
tracked entities in each copy. The resulting dataset has 100 tracked 
entities, i.e. 4,950 entity pairs, while on average 1,800 SDEs take 
place per sec. According to the use case survey of the Event Pro¬ 
cessing Technical Society [5], in the resulting dataset there are more 
SDEs per sec than in most applications. First, we used a single pro¬ 
cessor for CE recognition. In this case, the intervals of approximately 
64,000 CEs were computed and stored. Second, we used all eight 
processors of the computer in parallel. Consequently, each instance 
of RTEC running on a processor computed and stored the intervals 
of approximately 8,000 CEs. We emphasize that the input data was 
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Figure 2: Event Recognition for Public Space Surveillance. 


the same in all sets of experiments: each processor receives SDEs 
coming from all tracked entities—i.e. there was no SDE filtering to 
restrict the input relevant for each processor. We rely only on the 
indexing mechanism of RTEC to pick out relevant SDEs from the 
stream. RTEC employs a very simple indexing mechanism: it merely 
exploits YAP Prolog’s standard indexing on the functor of the first 
argument of the head of a clause. 

As in the previous set of experiments, the inter-query step is set 
to 5 sec, while the size of the WM varies from 10 to 110 sec. In this 
case, however, step includes approximately 9,000 SDEs, and WM 
varies from 18,000 to 192,000 SDEs. Figure 2(b) shows the aver¬ 
age CE recognition times. In all cases RTEC performs real-time CE 
recognition. Figure 2(b) also shows that we can achieve significant 
performance gain by running RTEC in parallel on different proces¬ 
sors. Such a gain is achieved without requiring SDE filtering. 

5 Discussion 

We presented RTEC, an Event Calculus dialect with novel implemen¬ 
tation techniques that allow for efficient CE recognition, scalable to 
large numbers of SDEs and CEs. RTEC remains efficient and scal¬ 
able in applications where SDEs arrive with a (variable) delay from, 
or are revised by, the SDE detection systems: it can update the al¬ 
ready recognised CEs, and recognise new CEs, when SDEs are arrive 
with a delay or following revision. 

RTEC has a formal, declarative semantics as opposed to most 
complex event processing languages, several data stream processing 
and event query languages, and most commercial production rule 
systems. Furthermore, RTEC has available the power of logic pro¬ 
gramming and thus supports atemporal reasoning and reasoning over 
background knowledge (as opposed to e.g. [2, 13, 19, 9]), has built- 
in axioms for complex temporal phenomena (as opposed to [26, 1]), 
explicitly represents CE intervals and thus avoids the related logical 
problems (as opposed to e.g. [22, 13, 9, 15]), and supports out-of- 
order SDE streams (as opposed to [14, 12, 9, 11, 20, 24]). Concern¬ 
ing the Event Calculus literature, a key feature of RTEC is that it 
includes a windowing technique. In contrast, no Event Calculus sys¬ 
tem (including e.g. [8, 7, 23, 24, 6]) ‘forgets’ or represents concisely 
the SDE history. 

The ‘Cached Event Calculus’ [8] performs update-time reason¬ 
ing: it computes and stores the consequences of a SDE as soon as 
it arrives. Query processing, therefore, amounts to retrieving the ap¬ 
propriate CE intervals from the computer memory. When a maximal 


interval of a fluent is asserted or retracted due to a delayed SDE, the 
assertion/retraction is propagated to the fluents whose validity may 
rely on such an interval. E.g. propagateAssert{[Ti, T 2 ], U) in the 
Cached Event Calculus checks whether there are new initiations as 
a result of asserting the interval (ri,r 2 ] of fluent U. In particular, 
propagateAssert checks whether: (1) the asserted fluent (7 is a con¬ 
dition for the initiation of a fluent F at the occurrence of event E, 
(2) the occurrence time T of E belongs to (Ti, r 2 ], and (3) there is 
not already a maximal interval for F with T as its starting point. If 
the above conditions are satisfied, propagateAssert recursively calls 
updateInit{E, T, F) in order to determine if F is now initiated at 
T, and if it is, to update the fluent interval database accordingly. 

propagateAssert also checks whether there are new terminations 
as a result of a fluent interval assertion, while propagateRetract 
checks whether there are new initiations and terminations as a re¬ 
sult of a fluent interval retraction. The cost of propagateAssert 
and propagateRetract is very high, especially in applications where 
the CE definitions include many rules with several fluents that de¬ 
pend on several other fluents. Furthermore, this type of reasoning 
is performed very frequently. RTEC avoids the costly checks ev¬ 
ery time a fluent interval is asserted/retracted due to delayed SDE 
arrival/revision. We found that in RTEC it is more efficient, and 
simpler, to discard at each query time Qi, all intervals of flu¬ 
ents representing CEs in {Qi— WM, Qi] and compute from scratch 
all such intervals given the SDEs available at Qi and detected in 
{Q,-WM,Q,]. 

For further work, we are developing techniques, based on 
abductive-inductive logic programming, for automated generation 
and refinement of CE definitions from very large datasets, with the 
aim of minimising the time-consuming and error-prone process of 
manual CE definition construction [16]. We are also porting RTEC 
into probabilistic logic programming frameworks, in order to deal 
with various types of uncertainty, such as imperfect CE definitions, 
incomplete and erroneous SDE streams [17]. 
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